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Foreword 


In this publication ENQA has taken the opportunity to bring together two distinct but 
related reports. 

Part 1 examines the language of European quality assurance. It grew out of the 
debates and discussions at an ENQA workshop in Warwick in June 2006. The success 
of the workshop highlighted the support needed by all agencies as they deal with the 
linguistic challenges involved in creating a European Higher Education Area (EHEA). 

Part 2 contains the findings of the second Quality Convergence Study (QCS II). 

The first Quality Convergence Study (QCS) examined why national quality assurance 
systems operate in different, yet commonly identifiable, ways in particular national 
contexts. Building on this, QCSII has sought to go beyond the technical language of 
quality assurance and provide an opportunity for agencies to reflect on the values that 
underpin their quality assurance systems. 

Both of these reports aim to contribute to promoting understanding of the multiple 
layers of meaning which become apparent when we attempt to understand quality 
assurance across borders, and across languages. They do not offer a conclusion, but 
a point of departure for further research and reflection. I hope that ENQA members 
and other stakeholders will find the points they raise both informative and thought- 
provoking. 


Peter Williams 
President, 

ENQA 


Introduction 


With the process of rapid change in the field of European quality assurance over the 
last decade, and in particular with the increase in speed since the start of the Bologna 
process, a great deal of research has been conducted on all areas of the field. Whilst 
useful in itself, much of this research has demonstrated the need for further work, 
especially in the field of communication. 

A wide range of actors, from independent research agencies, to academics, to 
stakeholders large and small, have endeavoured to contribute to the discussion and add 
evidence to the debate. The breadth of the research conducted and the information now 
available to all players can only help further the goal of creating a European Higher 
Education Area (EHEA). 

The majority of research has, however, taken an approach which, perhaps, has had 
to bypass some of the underlying issues present when dialogue is established across 
national, language and even individual agencies’ boundaries. 

An acknowledgement of the existence of personal subjectivity and the social 
construction of ideas in our understanding of European quality assurance may, to some, 
be seen as a distraction from the concrete work of bringing about the EHEA. This 
would, however, be a counter-productive position to hold. Quality assurance is carried 
out by individuals and groups of individuals that are situated in cultural and linguistic 
systems which have an impact on everything they do. An understanding of these sub 
contexts is vital for the successful creation of the EHEA at a more profound level. 

In 2006 ENQA began to address the challenges of communication and language in 
quality assurance in two ways. This publication brings together two separate reports 
that both, in their own particular ways, aim to contribute to deepening the discussion 
in European quality assurance. 

The first, the report of an ENQA workshop, deals with the language and terminology 
that we use everyday in national quality assurance contexts, some of which has been 
incorporated into European level documents such as the Standards and Guidelines for 
Quality Assurance in the European Higher Education Area (ESG). What do these terms 
‘mean’ across different linguistic and national contexts? The majority of people working 
with the European quality assurance are not working in their native language, but in 
English. In addition, there is no guarantee that those who are native English speakers 
necessarily always mean the same things by the same terms. 

The second, the report of the Quality Convergence Study II (QCS II), builds on this 
understanding to look at the terms in use which go beyond the “technical” language of 
quality assurance, and actually refer to deep rooted notions which are often taken for 
granted. The QCS II project sought to dig deeper and examine the values that underpin 
the terms we use, often implicitly, and what these say about the entire undertaking of 
the quality assurance of higher education. 

In the context of this work on matters to do with intercultural communication, 
the QCS II report was designed, written and approved simultaneously in English and 
French. The publication of both versions allows the bilingual reader to experience in 
a concrete way the inherent difficulties of working in a multicultural context and of 
providing an accurate translation. It also highlights the richness that such a way of 
working can bring to a project of this type. 


Part 1. 


The Language of European Quality 
Assurance - report from the ENQA 
workshop of 29-30 June 2006 

Rachel Dearlove, QAA 
Emmi Helle, ENQA 

1. Introduction 

The ENQA workshop on the language of European quality assurance, hosted by the 
Quality Assurance Agency (QAA, UK) in Warwick, revealed a wide range of points 
of interest, discussion and challenge for all those attempting to work across language 
boundaries to translate, understand and implement quality assurance ideas, processes 
and procedures. 

The workshop brought together representatives from a wide range of agencies, 
representing 16 different countries and at least 12 different languages. The discussion 
focused predominantly on the use of English as a ‘mediating’ language for the 
European quality assurance community and the impact that this has on the clarity of 
communication. The intent of the workshop was not to produce a ‘glossary’ of quality 
assurance words but to open up the debate on language and raise awareness of the 
problems and pitfalls of working across language boundaries. 

2. Word clusters 

The discussions were structured around four groups or ‘clusters’ of words. 

2.1 CLUSTER A: QUALITY, STANDARDS, QUALITY ASSURANCE, QUALITY 
CONTROL, ACCOUNTABILITY, ENHANCEMENT, IMPROVEMENT 

The word ‘quality’ was universally felt to lie at the heart of the work of all the 
participants, and indeed featured heavily in the names of the agencies they represented. 
By its very nature, however, it was felt to be very hard to pin down to a definition in any 
language. A distinction was made between the absolute definition of quality - which 
would allow ranking of outcomes on a scale - and the relative definition of quality 
which involves judgement against a set standard. In many cases it was not clear if this 
distinction was made within languages let alone between languages. 

It emerged that the use of the word ‘standard’ was open to a wide range of 
interpretations across languages. In the UK the word refers to a level of achievement 
measured against a reference point. In France and Germany its translation is used 
to indicate an average or norm. The word ‘criterion’ was suggested as analogous to 
standard although it was clear that this was not universally accepted. In relation to 
the European Standards and Guidelines (ESG) the word had been interpreted by the 
authors as meaning ‘principles’ to adhere to rather than something to be measured 
against. 

‘Quality assurance’ is a term imported into higher education from the world of 
business (and predominantly from the sector of manufacturing) as is the related term 


‘quality control’. In France ‘quality management’ has been used often in place of ‘quality 
assurance’ in the translation of the ESG, with the intention of instilling a sense of 
responsibility to the academics to manage quality for themselves. An interesting point 
was raised by a Russian delegate indicating that in their system the term following 
‘quality’ depended on the audience addressed i.e. quality assurance for broader society, 
quality control for regulators and quality enhancement for the higher education sector. 

The discussion concerning ‘accountability’ showed that the definition of the word 
is dependent on the outcome of the question ‘accountable to whom?’ A higher 
education institution is accountable financially to its sources of funding, to students 
and the general public as ‘consumers’ of higher education and its products (graduates), 
the government who are often commissioners (and sponsors), and the academic 
community. The term has both economic and ethical interpretations. However, in 
Finland and Russia, for example, the economic connotation of ‘accountability’ is 
weaker, and it could be replaced by ‘responsibility’. UK participants also described the 
range of interpretations of the word in terms of the level of accountability, ranging 
from: ‘tell us what you do’ to ‘explain what you do’ to ‘justify what you do’ - showing the 
range of meaning that ‘accountability’ can have. The term also implies that there is an 
external party to whom one is accountable. 

‘Enhancement’ and ‘improvement’ have very similar dictionary definitions in 
English as well as in other European languages and are often used interchangeably 
as synonyms. Among the participants, however, there was evidence of a definite 
preference for the use of the word ‘enhancement’ (and its translations) or ‘quality 
development’. ‘Improvement’ was felt to be more connected to advertisement and was 
perceived as more an aggressive term than that of ‘enhancement’. 

2.2 CLUSTER B: PRINCIPLES, VALUES, INDEPENDENCE, AUTONOMY, 
ACADEMIC FREEDOM, COMPLIANCE, CONFORMITY, ADHERENCE, 
CONVERGENCE, HARMONISATION, STANDARDISATION 

Principles, values, independence, autonomy and academic freedom all arise from 
ethical discourse. In particular principles/values were felt to imply a general moral 
obligation or duty. From an English definition point of view, there are subtle differences 
between the understanding of ‘independence’ and ‘autonomy’ but these differences 
were not reflected clearly in the understanding of the words by participants. As with 
‘accountability’, the word was felt to be defined by the relationship it implies between 
actors i.e. independence from what, autonomy in relation to whom? It was asked 
whether an agency was independent if it received funding from, or its staff members 
were appointed by, the governments or HEIs. It was felt, however, that the most 
important thing for the QA agencies was to be operationally independent, so that no 
external party could influence their decisions. It was questioned whether institutional 
autonomy, in any real sense, is compatible with external quality assurance. 

Academic freedom is, of course, a term that exists almost exclusively in a higher 
education context and usually stands for freedom from external control and influence. 

It is also understood as freedom of students and of the teaching staff to pursue their 
academic activities. Four pairs of concepts were suggested to help circumlocate its 
meaning: 


• Self-regulation vs. external inspection 

• Academic community vs. corporate business enterprise 

• Deliberative debates vs. executive decision 

• Academic democracy vs. managerialism 

It was suggested that ‘peer review’ might be a negotiated balance between external 
regulation and academic freedom. 

UK participants explained the subtle distinction between the use of the terms 
‘compliance’, ‘conformity’ and ‘adherence’ (although this was not fully accepted by all 
native English speaking participants): 

• Compliance - connotations of external control/compulsion 

• Conformity - following normative expectations 

• Adherence - polite phrase to soften the blow 

Similar issues of the ‘acceptability’ of terms were raised in relation to convergence, 
harmonisation, and standardisation. The majority of participants reported that 
‘harmonisation’ was principally used in a European context and as such was rarely 
used to refer to processes at a national level. Standardisation was seen to have negative 
connotations by many participants - it signalled the end of diversity and autonomy. 
Convergence conjured up an image of different systems moving together to a single 
point. 

2.3 CLUSTER C: ASSESSMENT, EVALUATION, INSPECTION, 

ACCREDITATION, AUDIT, REVIEW 

UK participants explained that in their national contexts, ‘assessment’ normally refers 
to student assessment and the word is not regularly used outside this context. 

‘Inspection’ was not widely used in any European country in a higher education 
context as it had connotations of more invasive procedures than are currently used in 
a quality assurance context. In particular it was felt to indicate the absence of peer 
review and always involve direct observation. ‘Audit’ and ‘review’ were fairly peculiar 
to the UK context where as ‘evaluation’ and ‘accreditation’ were more commonly in use 
across continental Europe. 

Participants discussed the relationship between the words as some felt that they 
fitted into a time sequence - with accreditation following on from an evaluation 
process. They also ranked the terms into soft (evaluation and review), middle (audit and 
assessment) and hard (inspection and accreditation). 

2.4 CLUSTER D: EXPERTS, PEERS, SELF-ASSESSMENT/EVALUATION, VISIT, 
REPORT, JUDGEMENTS, RECOMMENDATION, FOLLOW-UP 

Although the term ‘peer’ was known to all participants, in many cases the word 
was used interchangeably with ‘expert’ - a situation which is not the case in the 
UK. Participants agreed that both terms give a sense of seniority and to a certain 
extent authority - you have to have expertise to be an expert, and standing among 
the academic community to be considered a peer. Peers are likely to be experts in a 
particular subject area, whereas an individual with expertise may not necessarily be 
considered a peer. 


UK participants reported the shift in terminology from self-assessment to self- 
evaluation as a way of encouraging institutions to move from a ‘checklist’ to a more 
reflective account of their quality management. The change did not however, arise from 
any significant difference in the dictionary definitions of the words. It was pointed out 
that in French there is a problem translating any term including the word ‘self’ as this 
cannot be directly translated - the term ‘retrospective analysis’ is used in its place. 
Alternative, more ‘neutral’ words, suggested included ‘submission’ and ‘document’. In 
Finnish, for its part, there is no difference between self-assessment and self-evaluation. 
Participants from the Netherlands found that the self-evaluation regarded the writing of 
the report by the agency/institution to be evaluated; while assessment was the exploring 
of that report by an external evaluation team. 

The word ‘visit’ seemed to normally be prefixed by an additional describer i.e. 
evaluation visit, audit visit. The most common interpretation was that a ‘visit’ implied 
actually going somewhere, in most cases visiting the institution. UK participants 
introduced an additional and related term, ‘event’, which has a particularly unclear 
definition - a visit can include an event, and an event can include a visit! The nature of 
a ‘visit’ can also be qualified by whether it is a formal or informal ‘visit’. UK colleagues 
also highlighted that while across Europe a ‘site visit’ implies a visit to an institution 
involving any of a number of activities; in the UK ‘site visit’ implies a visit to specifically 
look at physical facilities or resources. 

Although ‘report’ was a word which was felt to be commonly understood it was noted 
that the Shorter Oxford English dictionary actually gives 15 different definitions of the 
term. Key associations for participants were that it was: a formal account following an 
investigation, which can include judgements or assess achievements. 

‘Judgement’ was universally felt to be a powerful word, one that did not always reflect 
the true nature of the outcome of the external quality assurance process. Alternatives 
used included ‘the decision’ (to accredit/not accredit), ‘findings’, ‘recommendations’ (see 
below) and ‘expert statement’. 

‘Recommendation’ as a term was felt to have varying ‘weight’ depending on whether 
the recommendations imposed conditions on the institution under scrutiny. It was 
contrasted by some participants with ‘points for consideration’. Recommendation was 
felt to imply a position of expertise and authority on the part of the agency, which might 
not be well received by institutions. 

The term ‘follow up’ was felt to contain a wide range of possible meanings, depending 
on both the system in which it was used and the context within that system. In all cases 
it was thought to be a continuation or repetition of an action, but it could be: 

• required or voluntary 

• formal or informal 

• timebound or not 

• evaluative or not 

• an action of the agency or of the institution 

For example, in Finland ‘follow up’ is always used to mean a follow-up evaluation by the 
agency of an institution; whereas in Germany it refers to what happens in an institution 
after a review. 


3. TAKING THE DEBATE FORWARD 

The workshop provided a welcome forum for debate and it was felt by many participants 
that the discussion should continue in some form. It also became clear that more 
words than those that had already been discussed caused confusion and raised points 
of contention and so should be admitted to the discussion. These new words have been 
added to the before mentioned word clusters in Annex III to this report. 

This report is intended to provide a source of information and a basis for those 
who were not present at the workshop to contribute to a further discussion. It was 
initially planned that this discussion would continue through an online message board, 
established by ENQA in September 2006. The contents of the board were intended to 
be analysed and to feed back to the current report. However, the online board did not 
attract the attention of ENQA members in a planned way. This might have been due 
to the time constraints and other projects in which the active ENQA member agencies 
were committed to. It is however worth wile experimenting different participative 
fora where discussions can be led between the ENQA member agencies and other 
stakeholders. It is obvious that an attempt to create further discussion can always be 
either successful or less successful, but this cannot be known if one does not have 
a try. ENQA will thus continue to take different initiatives on quality assurance and 
the important discussion on the language and terminology of quality assurance will 
definitely be continued by ENQA in different forms and fora. 


Annex I: Programme of the Language 
Workshop 

The Quality Assurance Agency for Higher Education 

The Language of European Quality Assurance 
An ENQA workshop 
29-30 June 2006 

Scarman House, University of Warwick, Coventry, UK 

Many of the difficulties in communicating ideas, processes and procedures about 
quality assurance amongst ENQA members result from both the number of languages 
used by them and, in particular, the consequences of English having been adopted 
as the common international language for the purpose. Many terms used in quality 
assurance do not easily translate from English into other European languages and vice 
versa. The use of English as the ‘mediating’ language between others can also give rise 
to misunderstanding and even a misplaced sense of effective communication. 

There have frequently been calls for the development of authoritative and 
comprehensive ‘glossaries’ of concepts, words and terms used in quality assurance 
in Europe, but these have invariably proved to be of limited value because of the 
difficulty in agreeing the meanings of the concepts and words themselves in English. 
The consequent lack of a shared linguistic understanding is one of the major stumbling 
blocks to the full development of the European Higher Education Area, at least in terms 
of shared values for quality assurance. 

Many people are involved in the translation of documents into and out of English 
in the course of their work for quality assurance agencies. Some is undertaken by 
the quality assurance ‘experts’ and agency leaders as part of their normal activities. 

For them the problem maybe limited, as they have opportunities to discuss their 
work in English during the course of their international activities and to develop an 
understanding of the complexities of other systems and the way these are expressed 
through English. Some is the responsibility of professional translators and interpreters 
who do not, generally, get these opportunities. 

This ENQA workshop is aimed at anyone whose work requires them to translate 
technical documents relating to quality assurance in higher education into or out of 
English. 

The aims of the workshop are to: 

• improve understanding and communication amongst ENQA members by 
examining English words and phrases commonly used in quality assurance; 

• facilitate future attempts to develop glossaries of those words and phrases. 

The objectives of the workshop are to: 

• help participants to gain a confident understanding of the range of meanings of 
relevant English words and phrases used in quality assurance and to relate them 
more firmly to the corresponding words and phrases in their own languages 

• place these words and phrases in their theoretical and operational contexts. 


The workshop will be interactive, with opportunities for participants to share the 
challenges they face in translating quality assurance concepts and terms. They will be 
invited to add words to the clusters indicated in the programme. It will be led by Peter 
Williams (CEO of QAA and President of ENQA), and facilitated by Professor Susan 
Bassnett (Pro Vice-Chancellor and Professor of Comparative European Literature, 
University of Warwick), Carolyn Campbell (Assistant Director QAA), Fiona Crazier 
(Assistant Director QAA) and Dr Peter Findlay (Assistant Director, QAA). 

FINAL PROGRAMME 

THURSDAY, 29 JUNE 2006 
10:30 Registration 

11:00 Introduction: The scope of the workshop 

11:20 Word cluster A: the basic concepts 
quality 
standards 

quality assurance/quality control 
accountability 

enhancement/improvements 
13:00 Lunch 

14:00 Word cluster B: principles 
principles/values 

independence/autonomy /academic freedom 
compliance/conformity /adherence 
convergence/harmonisation/standardisation 

15:00 Break 

15:30 Groups to discuss questions arising for them from the previous sessions 

16:30 Word cluster C: types of quality assurance: 
assessment 
evaluation 
inspection 
accreditation 
audit 
review 

18:00 Day ends 
19:30 Workshop dinner 


FRIDAY, 30 JUNE 2006 


°7 : 3°- Breakfast 
08:30 

09:00 Word Cluster D: operational processes and procedures: 
experts/peers 

self-assessments/self evaluation 

visits 

reports 

judgements 

follow-up 

recommendation 

10:30 Break 

11:00 Groups to discuss questions arising for them from the previous session 
12:00 Final session: outstanding questions and summary 
13:00 Lunch and depart 


Annex II: List of participants of 
the workshop on the Language of 
European Quality Assurance 


Tiia Bach 

Expert, 

SA Archimedes, Estonian Higher Education Accreditation 
Centre, Estonia 

Esther Balboa 

Technical Staff, Accreditation Programme 

ANECA - Agencia Nacional de Evaluacion de la Calidad y 

Acreditacion, Spain 

Susan Basnett 

Pro-Vice Chancellor, University of Warwick and 
Member of the Board, QAA - Quality Assurance Agency for 
Higher Education, UK 

Carolyn Campbell 

Head of International Affairs, Chief Executive’s Group, 

QAA - Quality Assurance Agency for Higher Education, UK 

Vera Chepurnykh 

Project Manager, 

National Accreditation Center of the Russian Federation, 
Russia 

Pierre Couraud 

Project Manager, 

CNE - Comite National devaluation, France 

Fiona Crozier 

Assistant Director, Development and Enhancement Group, 
QAA - Quality Assurance Agency for Higher Education, UK 

Bruno Curvale 

Project Manager, Representative of ENQA Steering Group, 
CNE - Comite National devaluation, France 

Julie Davies 

Policy Development Manager, 
Association of Business Schools, UK 

Rachel Dearlove 

Aide to the Chief Executive, Chief Executive's Group, 

QAA - Quality Assurance Agency for Higher Education, UK 

Peter Findlay 

Assistant Director, Reviews Group, 

QAA - Quality Assurance Agency for Higher Education, UK 

Patricia Georgieva 

Chief Executive, 

National Evaluation and Accreditation Agency, Bulgaria 


Emmi Helle 

Project Officer/ Acting Secretary General, 

ENQA European Association for Quality Assurance in Higher 
Education, Finland 

Fabrice Henard 

Project Manager, Representative of QCS project 
CNE - Comite National devaluation, France 

Achim Hopbach 

Managing Director, 

German Accreditation Council, Germany 

Trine Hornemann 

Language Consultant, 

EVA - The Danish Evaluation Institute, Denmark 

Volker Husberg 

Consultant, 

Agentur fair Qualitatssicherung durch Akkreditierung von 
Studiengangen - AQAS e.V, Germany 

Sue Ibberson 

Conference Administrator and PA to the Chief Executive and 
Head of International Affairs, 

QAA - Quality Assurance Agency for Higher Education, UK 

Mercedes Jaime 

External Advisor for Institutional and International Relations, 
ANECA - Agencia Nacional de Evaluation de la Calidad y 
Acreditacion, Spain 

Mark Kalinin 

Member of the Board 

Agency for Higher Education Quality Assurance and Career 
Development, Russia 

Helka Kekalainen 

Senior Advisor, 

FINHEEC - Finnish Higher Education Evaluation Council, 
Finland 

Teresa Lee 

Programme Manager, 

Irish Universities Quality Board, Ireland 


Bianka Lichtenberger Head of Accreditation, 


Thierry Malan 

FIBAA - Foundation for International Business 
Administration Accreditation, Germany 

General Inspector of the Administration of National Education 
and Research, General Inspectorate of the Administration of 
National Education and Research, IGAENR, France 

Barbara Michalk 

Programme Manager, 

German Rectors’ Conference, The Quality Assurance Project, 
Germany 


Sirpa Moitus 

Project Manager, 

FINHEEC - Finnish Higher Education Evaluation Council, 
Finland 

Gintar Motiekaitien 

Chief Officer, 

Centre for Quality Assessment in Higher Education, Lithuania 

Ariane Nussbaum 

Scientific Collaborator, 

OAQ Center of Accreditation and Quality Assurance of the 
Swiss Universities, Switzerland 

Trish O’Brien 

Development Officer, 

National Qualifications Authority of Ireland, Ireland 

Karin Riegler 

Senior Programme Manager, 

European University Association, EUA, Belgium 

Vyacheslav Samoilov 

Director General, 

Agency for Higher Education Quality Assurance and Career 
Development, Russia 

Gerhard Schreier 

CEO, 

Evaluation Agency Baden-Wuerttemberg, Germany 

Kurt Sohm 

Managing Director, 
Fachhochschule Council, Austria 


Konstantinos Tzortzis Development Officer, International Collaboration, 


Els Van Zele 

Chief Executive’s Group, QAA - Quality Assurance Agency 
for Higher Education, UK 

Staff Member, Quality Assurance, 

VLIR - Flemish University Council, Belgium 

Michele Wera 

Policy Advisor, 

Accreditation Organisation of the Netherlands & Flanders 
(NVAO), The Netherlands 

Peter Williams 

President ofENQA, Chief Executive, 

QAA - Quality Assurance Agency for Higher Education, UK 


Annex III: Proposed new word 
clusters 

These word clusters have been re-arranged and expanded (new words are marked in 
italics) in the light of the workshop discussions. 

Cluster A: fundamentals 

• quality 

• quality assurance 

• quality control 

• enhancement 

• improvement 

Cluster B: ethical concepts 

• accountability 

• principles 

• values 

• independence 

• autonomy 

• academic freedom 

• transparency 

Cluster C: 'measurement' 

• standards 

• competence 

• specification 

• benchmark 

• criteria/on 

• guidelines 

• code of practice 

• objective 

• outcome 

• output 

Cluster D: descriptions 

• compliance 

• conformity 

• adherence 

• convergence 

• harmonisation 

• standardisation 

• comparability 

• equivalence 


Cluster E: methods 

• assessment 

• evaluation 

• inspection 

• accreditation 

• audit 

• review 

Cluster F: processes and procedures 

• expert 

• peer 

• visit 

• self-assessment/evaluation 

• report 

• document 

• submission 

• statement 

• judgement 

• follow-up 

• action plan 

• feedback/feedback loop 

• recommendation 

• conditions 


Part 2. 


Final report on the pilot Quality 
Convergence II project : 1 
Promoting epistemological 
approaches to quality assurance 

Fiona Crozier, QAA 
Bruno Curvale, CNE 
Fabrice Henard, CNE 


1. Introduction 

The Quality Convergence Study II (QCS II) built upon the successful outcomes of QCS 

I. QCS II was intended to provide further stimulus to ENQA member agencies in their 
discussions and implementation of capacity building. The main outcome from QCS I 
was the conclusion that, rather than, or perhaps before, the development of common 
methods of working, it was important to further develop a real understanding of the 
national contexts within which the quality assurance agencies work. There is a need 
to develop an understanding that goes beyond how the quality assurance agencies do 
things, and leads to a comprehension of why the agencies work as they do. 

QCS II operated within the context of a continuing debate on how agencies might 
work together, whether this is at the level of furthering mutual understanding or 
seeking mutual recognition of the outcomes of evaluations. It was an attempt to 
question the foundations of quality assurance in higher education. 

Initiatives for the further development of the European dimension of quality 
assurance (QA) in higher education (HE) such as the publication of the Standards and 
Guidelines for Quality Assurance in the European Higher Education Area (ESG), the 
TEEP II project and the Nordic Network project on the implications of implementing 
the ESG in the Nordic countries, were also taken under consideration in the QCS 

II. This was because these initiatives too, focussed on working together, or on the 
implications of a European-wide set of agreed principles. The project team also read 
with interest the outcomes of a survey on ENQA membership criteria, and noted the 
work being carried out by ECA member agencies in terms of mutual recognition of 
accreditation decisions. 

In particular, the pilot project considered the discussion which developed during the 
ENQA seminar on the language of quality assurance held at Warwick, in the United 
Kingdom, in June 2006. This seminar confirmed that, in addition to the problems 
which arise from the use of a single working language (English), it is necessary, in 
parallel, to examine and discuss the values, often implicit, that underpin the work done 
at the individual agencies. 

The idea that underpins QCS II is that only a deeper shared understanding of those 
values will help to overcome any misunderstandings. In the opinion of the project team, 
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the recognition of the importance of what lies beneath the surface in agencies’ work is 
essential to the development of the European Higher Education Area (EHEA). What is 
regarded as beyond question in one country may not be regarded as such in another. If 
it is implicit in the modus operandi of one agency and therefore not discussed, a second 
agency may simply believe that it is of no importance. Thus, in the case of processes that 
are, on the surface similar, there is an assumption that the outcomes of the processes 
are the same. Working relationships within ENQA are constantly confronted with these 
unspoken and intercultural dimensions. 

The QCS II project chose to work towards understanding the values that underpin 
QA, in order to help provide better insight and inform capacity building between and 
within agencies. 

2. Methodology and timescale 

This pilot project ran over a period of one year (January to December 2006). It was 
guided by two agencies, CNE and QAA, who set up a steering group composed of Fiona 
Crozier (QAA), Bruno Curvale (CNE) and Fabrice Henard (CNE). 

The method consisted of inviting the ENQA member agencies to react and respond to 
four notions 2 which are usually present in any discourse about QA in higher education: 
independence, peer review, transparency and results. The main phases of the method 
were: 

• the agreement of the four notions and the development of a questionnaire 
(attached at annex B) about those notions (April to June 2006), 

• a consultation period when the questionnaires were sent to all member agencies 
of ENQA (July to October 2006), 

• analysis of the completed questionnaires (October to November 2006) and 

• the writing of a report (November to December 2006). 

It was agreed that the results of the questionnaire should not be published - the aim 
of the project was not to debate whether or not responses were good or bad, right or 
wrong, nor to comment on perceived good practice, but to consider the conditions, 
feasibility and interest in an epistemological debate amongst QA practitioners. 

The steering group was supported in its work by four experts chosen for 
their academic background (Philosophy, History, Social Sciences) and because they 
represented different areas of Europe (Bulgaria, France, Portugal and UK). All are 
academics with some knowledge of QA in their national context. It was a deliberate 
decision not to have a QA expert on the panel; this role was fulfilled by the three 
members of the steering group. Working with such a group of experts proved to be a 
real success and greatly enriched the project. 

The input of the experts and their interest in the project was invaluable to 
the steering group. The experts agreed that the decision to focus on four notions 
(independence, peer review, transparency and results), and to ask agencies to 
concentrate on the values that underpinned those themes or notions in their context 
was a good approach. They were involved in the design of the questionnaire that 


For the purposes of this project, the word “notion” refers to a concept or idea that is common to many European models of QA in 
HE, for example, peer review. The word “value” is used to discuss the values that underpin the four notions on which the 
questionnaire was based. 


was subsequently sent to all agencies, and they assisted in the analysis of the results. 
The discussions between the experts and the steering group at the two meetings held 
were both interesting and challenging. It was very useful for the project group to be 
confronted with the alternative and challenging views of HE and QA in HE that were 
raised by the experts. 

The outcomes reported below should be viewed with caution: they emerged from the 
reactions to and reflections on the responses to the questionnaire of a small group of 
people (the experts and the steering group). 

3. The responses to the questionnaire 

Thirteen agencies and the evaluation unit of the European University Association 
responded to the questionnaire; this represents 47% of agencies that regularly 
participate in ENQA activities (the list of respondents is attached as annex A). 

The development of the questionnaire and the responses to it provided both steering 
group members and experts with an opportunity for reflection: this was also true for 
the experts. Mutual comprehension between QA agencies and HEIs needs to be worked 
at and projects of this type offer the opportunity to interact positively. 

For example, the experts brought to the project their own perceptions of QA. They 
did not only react to the agencies’ responses, but also to what they perceived to be 
missing or unsaid. The members of the steering group were thus confronted not only 
with what might be implicit within their own national context but also with what 
is unquestioned or implicit across agencies as a whole. This reinforces the idea that 
it is important for agencies to explain more clearly, to HE providers and the public 
in general, their objectives and their approach to quality through the processes they 
develop. 

The experts were surprised by some of the things that they perceived to be “unsaid” 
or unimportant, whereas for the steering group, sometimes these things were implicitly 
understood. It is often these implicit values that are used by agencies to underpin their 
argument. 

For example, some agencies did not mention that their methodologies are the result 
of a process of dialogue and consensus between them and their stakeholders, with 
the result that the experts thought that such agencies used their procedures as a 
justification for their actions. In fact, such debate is a positive feature of the modus 
operandi of the agencies concerned. 

A more systematic approach to discussion or reflection would allow agencies to 
improve their position in a national or international debate on QA in HE. Indeed, 
the experts were of the opinion that the responses to the questionnaires were often 
defensive and that the agencies should be more self-confident, as they felt that this is 
one of the conditions necessary for a robust debate between partners/stakeholders. 

The four notions proposed - independence, peer evaluation, transparency and 
results - are integral to the various models of QA found in the European context. The 
debate with the experts allowed for the questioning of these notions. 

3.1 INDEPENDENCE 

This notion attracted the most detailed responses. Independence is at the heart of the 
majority of agencies in terms of the justification of their legitimacy (an issue dealt 
with in detail later in this report). However, it is necessary to distinguish between 


independence and autonomy: an agency may be independent, in the sense that the 
judgements it makes formally on the basis of recommendations of its evaluation teams 
are independent judgements, and are not influenced by the views of stakeholders. But 
it can rarely operate autonomously from those to whom it provides a service - these 
include providers of higher education (institutions), potential and current students, 
government and employers. All these are the agency’s raison d’etre and it provides a 
service to them. 

In addition, many responses showed that the words “independence” and “autonomy” 
are used interchangeably, giving rise to a problem of vocabulary (see the parallel report 
on the ENQA seminar “Language of Quality Assurance”, Warwick, June 2006 for 
further detail). 

The responses also show that it is necessary to distinguish between the question of 
the legitimacy of an agency and the question of the acceptability of its results by others 
(universities, ministries etc). 

3.2 PEER REVIEW 

Whatever the process of QA used (audit, institutional or programme evaluation, 
accreditation. . .), peer review of some sort exists at the heart of every evaluation. It is 
an attempt to allow the evaluation to operate in the most objective and yet acceptable 
way possible. 

The notion of peer review is understood differently by different agencies each of 
whom has its own idea of what the term means. In certain cases, the peers can only be 
drawn from a pool of university professors, in others from the entire pool of personnel 
relevant to the evaluation in question. The participation of students in evaluation 
procedures raises more questions about the legitimacy of those who participate in the 
judgement-making and advisory processes. 

In the end, the way in which evaluation procedures apply peer review varies between 
agencies and depends on their concept of what is or is not acceptable. For example, in 
some cases, peers carry out the review but an agency officer writes or edits the final 
report. In others, it is the responsibility of the peer reviewers to write the report. 

The matter of objectivity is also crucial. Peer evaluation has many features in 
common with academic judgement, and it is for this reason that it cannot be rendered 
completely objective by making it subject to a set of rules. It is also possible that 
one panel of peers might reach a different conclusion to another panel on the same 
evaluation. This can lead to disputes regarding the outcomes of evaluations, or the use 
of appeal procedures. 

3.3 TRANSPARENCY 

Transparency was cited by some agencies as something that helped them to reflect on 
their professionalism. The more an agency is transparent, the more it feels professional. 

Interestingly the notion of transparency is not generally demonstrated through 
the discussion of principles, but more often through a technical description of the 
methods and procedures used. Transparency is the notion that most easily lends itself 
to technical discourse. 

This notion was also used by the agencies to demonstrate legitimacy. However, 
is transparency of function enough in itself to assure legitimacy in the eyes of 
stakeholders? The experts were of the opinion that transparency was not necessarily a 


positive notion. It is possible for the notion of transparency to be abused since claims 
can be made that procedures are straightforward and openly available for scrutiny, but 
are often then carried out subjectively. 

Some responses highlighted a lack of criteria to address this value. How does one 
judge the transparency of an agency, of an evaluation method, or of the result of 
an evaluation without the necessary criteria? On the other hand, does a lack of 
transparency necessarily mean that the process or the judgement is unreliable? 

Can one assume that there is a lack of criteria because agencies do not make the 
values that underpin transparency explicit or stable? If so, such a notion may risk 
losing its sense and place in QA in HE if there are no explicit and agreed criteria to 
demonstrate its importance. 

3.4 RESULTS OF EVALUATIONS 

This value was not discussed in detail in the responses. However, in some, a political 
dimension to results emerged. 

Agencies gain some of their legitimacy through their mission to carry out a certain 
task - be it to make accreditation decisions, carry out audits, make public the results 
of an evaluation at subject or institutional level, safeguard standards and so on. It is 
difficult for them to question this task or mission. It may also underline the fact that 
agencies usually act within a framework designed by others (i.e. Governments, rector’s 
conference etc.). 

3.5 CONCLUSIONS FROM THE OUTCOMES OF THE QUESTIONNAIRE 

To conclude this section of the report, the four notions chosen for the questionnaire 
led to responses which highlighted the authority/legitimacy of agency practices. 
Interestingly none of the responses suggested that the wrong notions had been chosen 
or that the agency concerned did not believe one of the notions to be important. Can we 
therefore deduce that the legitimacy of the work of agencies is focussed on these four 
notions and the values that underpin them? 

4. Legitimacy: a central matter 

The QCS I study showed that the systematic organisation of quality assurance 
mechanisms is the result of interaction between partners (agencies, higher education 
providers, ministers etc). It suggested that there is no single definition or concept of 
quality. Rather the enhancement of quality in HE is the result of interaction between 
stakeholders. 

The QCS II study, for its part, turned the thoughts towards the acceptability of the 
results of evaluations; that is, towards the conditions that must be fulfilled before the 
results of evaluations - be they recommendations, assessment or accreditation decisions 
- are recognised as well-founded and legitimate. This is an important question 
because evaluation results impact on the decision-making processes in individual higher 
education providers and also on the decisions made by other stakeholders. 

In their responses to the questionnaire, the agencies tried to show the validity of 
their work by mentioning the mandate given to them, and their endeavours to ensure 
that the judgements that they published were governed by justice and equity. They 
immediately focused on virtue - to do well what has to be done - and sought to 


demonstrate that they were reliable and credible by putting forward a definition of 
professionalism based on the notions of transparency, independence and utility. 

It is possible that this tendency towards justification was caused in part by the way 
in which the questionnaire was designed, but the experts did not believe that to be 
the case. Rather they saw it as a sign that the agencies lacked a certain amount of 
self-confidence. 

It may also be due to the fact that agencies are not used to questioning their own 
activities. They do not tend to challenge the frameworks that form the basis of their 
raison d’etre and to question the political decisions that impact on their work. They 
resolutely see themselves as operators of a set of procedures and defend themselves 
against encroaching on the territory of those that they see as political decision makers 
or the academic world. 

The experts, a little provocatively (which was the nature of the relationship between 
them and the project steering group) wish to question the agencies on that essential 
area of their work: the results of evaluations. They invite them to consider more 
closely the nature of the judgements that they make and the notions of objectivity and 
transparency, not just the political dimension of quality assurance. 

The pilot QCS II project calls on agencies to consider more carefully the most 
fundamental questions. 

5. Respect for the process is not enough 

Beyond the confines of QCS II, the matter of legitimacy is currently a topical one. It is 
of central importance for agencies. 3 

Just as agencies do their utmost to demonstrate their transparency and 
independence; it would seem that respect for such values is not enough to underpin 
legitimacy. The outcomes of evaluations, which play an important part in higher 
education, claim to be the result of technical procedures and thus, entirely objective. 
But is this really the case? 

A look at the ESG can help to clarify things: they refer to the conditions necessary 
for evaluation results to be legitimate. Part one of the ESG addresses the legitimacy of 
the objectives of evaluation and parts 2 and 3 address the legitimacy of the processes 
and the results. Despite that, does strict adherence to the ESG lead automatically to 
legitimate and acceptable decisions? Those who run evaluations know from experience 
that one can only evaluate something if one knows its purpose. Will the procedures 
associated with an evaluation always come up with the right result? Can something 
really be evaluated if its own objectives are not taken into account in the process? So, 
what really leads to legitimacy and acceptability of results? 

6. The political nature of evaluation 

Agencies cannot claim to work outside the political arena. Their results are used by 
decision-makers and higher education providers to steer higher education systems and 
to appraise their quality. If the agencies do not take this into account, don’t they 
risk, behind the fapade of independence, being exploited? Doesn’t this position make 
dialogue with HE providers more difficult? 
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Cf. the Council of Europe seminar in Strasbourg on 18th and 19th September 2006 which focussed on “quality assurance and its 
legitimacy in higher education: the role of universities and national authorities.” 


7. To take things further 

The limitations of a procedural approach, and the recognition of these limitations, 
oblige the agencies to reflect on the matter. The following scenarios could usefully form 
the basis of some systematic reflection on their part in order to reinforce the quality of 
the link that binds the agencies to their partners: 

• The agencies are part of many actors in the higher education system. They 
are independent but not, however, necessarily autonomous. So where does the 
balance lie? What is the necessary distance - needed for evaluations - between 
agencies and higher education providers (which is the translation of the notion of 
independence in external evaluation) and the proximity which will allow for an 
understanding of the higher education system within which individual institutions 
are operating? 

• Peer evaluation is supposed to guarantee the legitimacy of an evaluation since it 
means that the system is being evaluated by those who understand it. This notion 
is generally shared by policy makers and by the academic world. It is not, however, 
exempt from criticism, because it is not enough to guarantee the acceptability of 
results. So, how can we consider the link between those who are being evaluated 
and the agencies that carry the evaluations out? How can we discuss the link 
between agencies’ activities and their purpose? The question of legitimacy is a 
collective one - shouldn’t the debate therefore involve all parties with an interest 
in higher education? 

• It is difficult for agencies to think about evaluation without considering the aims 
of higher education. Aren’t they right, along with other parties in the higher 
education system, to ask themselves what sort of results that system really needs 
from evaluations? 

Finally, it seems sensible to continue the questions opened up by this project to avoid 
the notions of independence, transparency and objectivity playing too exhaustive a role 
in the development of methodologies. Thus agencies may, in the spirit of openness, be 
able to fully play their part in higher education systems. 


The project steering group wishes to thank the experts who provided them with so 
much help throughout the QCS II study: 
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1. Introduction 

Letude pilote Quality Convergence Study II (QCS II) s’inscrit dans la continuity des 
conclusions de la premiere etude sur la convergence europeenne dans le domaine du 
management de la qualite dans lenseignement superieur, QCS I. Cette derniere visait 
a renforcer les capacites de debat entre les agences membres d’ENQA et a developper 
leurs competences. Elle a notamment mis en exergue l’importance des contextes 
nationaux et leur influence sur le fonctionnement des agences. Letude soulignait 
l’interet d’approfondir la comprehension des contextes nationaux prealablement a la 
mise au point de methodes de travail communes. Ce besoin de comprehension va 
au-dela de la simple connaissance des savoir-faire (le comment) pour s’attacher aussi aux 
raisons (le pourquoi) conduisant les agences a agir comme elles le font. 

QCS II poursuit ce questionnement des agences sur leurs modes de travail, que ce 
soit pour approfondir la comprehension mutuelle ou pour aboutir a la reconnaissance 
des resultats d evaluation. 11 s’agit de s’interroger sur les fondements de l’assurance 
qualite dans lenseignement superieur. 

Les initiatives pour le developpement de la dimension europeenne de l’assurance 
qualite, telles la publication des references europeennes en matiere de management 
de la qualite, letude Trans-European evaluation project (TEEP II) sur revaluation des 
programmes Erasmus Mundus ou celle du reseau des agences scandinaves portant sur 
les implications nationales des references europeennes, ont ete prises en consideration, 
car elles aussi ont mis l’accent sur les fa^ons de travailler ensemble et sur les 
implications des grands principes partages a l’echelle europeenne. Le groupe de pilotage 
s’est egalement interesse aux resultats dune etude interne sur les criteres d’adhesion a 
ENQA. 11 s’est aussi penche sur le travail mene par ECA en termes de reconnaissance 
mutuelle des resultats des processus d’accreditation. 

Le groupe de pilotage a porte une attention particuliere aux debats menes lors du 
seminaire ENQA sur la terminologie de l’assurance qualite (Warwick, juin 2006). Ce 
seminaire a confirme qu’au-dela d’une reflexion sur les problemes de comprehension 
souleves par l’utilisation d’une langue de travail unique (l’anglais), il est necessaire 
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Ce rapport a ete redige parallelement en langues anglaise et fran^aise. Il a ete valide dans les deux langues. 


d’avoir, en parallele, un debat sur les valeurs, souvent implicites, qui fondent les 
activites des agences. 

L’idee qui sous-tend QCS II est que seule une meilleure connaissance partagee de 
ces valeurs permettra de depasser certaines incomprehensions. Le groupe de pilotage 
juge essentielle l’identification des dimensions implicites du travail des agences : ce 
qui pose question dans tel pays ne s’envisage peut-etre pas de la meme fagon dans tel 
autre. Quand la dimension implicite n’est pas objet de debat, les pratiques dune agence 
n’induisent pas forcement de questionnement de la part des autres agences. Ainsi, une 
agence peut considerer a tort, en raison de la similitude des methodes, que les finalites 
poursuivies par une autre agence sont les memes que les siennes. Les relations de 
travail au sein d’ENQA sont en permanence confrontees a ces dimensions implicites 
et interculturelles. 

2. Methodologie et calendrier 

Cette etude est un projet pilote qui s’est deroule sur un an (de janvier a decembre 2006). 
11 a ete conduit par deux agences (CNE et QAA) qui ont constitue un groupe de pilotage 
compose de Fiona Crozier (QAA), Bruno Curvale (CNE) et Fabrice Henard (CNE). 

La methode retenue a ete d’inviter les agences d’ENQA a exprimer leurs points 
de vue et a reagir sur quatre notions 5 habituellement utilisees dans le domaine de 
l’assurance qualite : l’independance, revaluation par les pairs, la transparence et les 
resultats. 

Les principales etapes de la procedure ont ete les suivantes: 

• elaboration du questionnaire, presente en annexe B (d’avril a juin 2006) ; 

• consultation de l’ensemble des agences membres d’ENQA (de juillet a octobre 
2006) ; 

• analyse des reponses au questionnaire (d’octobre a novembre 2006) ; 

• redaction du rapport (de novembre a decembre 2006). 

11 a ete decide que les reponses du questionnaire ne seraient pas publiees. L’objectif 
de l’etude n’etait pas de debattre sur le fait de savoir si les reponses etaient correctes 
ou non, ni de reperer des bonnes pratiques. 11 s’agissait d’analyser les conditions, 
la possibility et 1’interet d’un debat de nature epistemologique entre praticiens de 
l’assurance qualite. 

Le groupe de pilotage a travaille en collaboration avec quatre experts originaires 
de quatre pays differents (Bulgarie, France, Portugal et Royaume-Uni) et choisis 
pour leurs competences academiques (philosophic, histoire, sciences sociales). Ces 
quatre universitaires connaissaient les systemes d’assurance qualite de leur pays et ont 
beaucoup enrichi le projet. 11 a ete decide de ne pas recruter d’expert en assurance 
qualite (ce role etait assure par les trois membres du groupe de pilotage). 

Les apports de ces experts et leur interet pour cette etude ont ete precieux au 
groupe de pilotage. Ils ont confirme 1’interet de l’approche centree sur les notions 
d’independance, devaluation par les pairs, de transparence et de resultats et sur une 
demande aux agences de s’interroger sur les valeurs qui etayent ces notions en fonction 


5 Dans le cadre de ce projet, le mot notion renvoie a une idee ou a un element present dans de nombreux modeles europeens de 
l’assurance qualite de l’enseignement superieur, par exemple : revaluation par les pairs. Le mot valeur renvoie ici aux idees 
utilisees par les agences pour justifier les notions proposees a leur reflexion dans le questionnaire. 


de leur propre contexte. Les experts ont ete associes a l’elaboration du questionnaire 
et ont contribue a l’analyse des reponses. Les discussions entre eux et le groupe de 
pilotage ont ete stimulantes. Ce dernier a juge tres interessant d’etre confronte aux 
points de vue alternatifs et provocateurs des experts sur l’enseignement superieur et 
l’assurance qualite. 

Les considerations rapportees ci-dessous n’engagent pas les institutions dont sont 
membres les auteurs. Elies resultent des reactions et des reflexions de ce petit groupe 
de personnes. 

3. Les reponses au questionnaire 

Treize agences, ainsi que l’unite d evaluation institutionnelle de l’EUA ont repondu au 
questionnaire, soit 47 % des organismes qui participent regulierement aux activites 
d’ENQA (la liste est presentee en annexe A). 

La mise au point du questionnaire et les reponses correspondantes ont offert 
des occasions de reflexion pour le groupe de pilotage comme pour les experts. 

La comprehension mutuelle entre les agences et les etablissements d enseignement 
superieur a besoin d’etre pensee en commun. Des projets de ce type en fournissent 
l’occasion. 

C’est ainsi que les experts ont enrichi l’etude de leurs propres perceptions de 
l’assurance qualite ; ils ont reagi aux reponses des agences, mais ont aussi exprime 
leurs opinions sur ce qui manquait ou ce qui relevait du non-dit. A cette occasion, le 
groupe de pilotage a compris que les agences se trouvaient confrontees non seulement 
aux dimensions implicites qui perturbent les relations qu’elles entretiennent entre 
elles, mais aussi a une part d’implicite commune a toutes mais qu’elles ne partagent 
pas necessairement avec leurs autres partenaires. Ce constat renforce l’idee qu’il est 
important de mieux expliquer aux etablissements d’enseignement superieur et a leurs 
publics les objectifs et les approches des demarches qualite developpees par les agences. 

Les experts ont ete surpris par le nombre de reponses qui, selon eux, ne traitaient 
pas certains aspects, alors que pour le groupe de pilotage ces aspects etaient tellement 
evidents qu’ils n’avaient pas a etre mentionnes. Les agences interrogees ont souvent 
employe ces valeurs implicites pour etayer leurs arguments. 

Ainsi, certaines agences oublient de rappeler, en presentant leurs methodes, que 
ces dernieres sont le resultat d’un dialogue et d’un consensus avec leurs partenaires. 

11 en resulte que les experts ont considere que les agences mettaient en avant leurs 
procedures pour justifier leur action et se proteger, alors meme que leur demarche 
traduisait en fait une reelle ouverture vers les partenaires et les etablissements. 

Une approche plus methodique des discussions ou des reflexions permettrait aux 
agences de conforter leur position dans les debats nationaux ou internationaux sur 
l’assurance qualite. En effet, les experts ont estime que les reponses au questionnaire 
etaient souvent defensives et qu’il serait dans l’interet des agences d’acquerir plus de 
confiance en elles-memes. C’est, selon eux, une des conditions de la vigueur du debat 
entre partenaires dans l’enseignement superieur. 

Les quatre notions proposees (independance, evaluation par les pairs, transparence 
et resultats) font reference aun modele de procedures d’assurance qualite. Le debat 
avec les experts fut l’occasion d’examiner ces notions, caracteristiques du modele 
europeen d’assurance qualite. 


3.1 INDEPENDANCE 

Cette notion a suscite les reponses les plus detaillees. L’independance est au coeur 
des preoccupations de la majorite des agences quand elles cherchent a justilier leur 
legitimite. 11 est necessaire id de distinguer independance et autonomie : une agence 
est independante dans le sens ou les jugements quelle formule sur la base formelle des 
recommandations des equipes d’experts sont independants et non influences par les 
points de vue des parties prenantes. En revanche, une agence peut rarement operer de 
maniere autonome par rapport a ceux a qui elle fournit un service - les etablissements, 
les etudiants (presents ou a venir), les gouvernements et les employeurs. Ce sont ces 
groupes qui justifient l’existence meme de l’agence, dont la mission est de leur rendre 
ce service. 

En outre, de nombreuses reponses revelent que les mots « independance » et 
« autonomie » sont employes de maniere interchangeable pour signifier l’une ou l’autre 
notion, demontrant par la un probleme de vocabulaire. 

Enfin, les reponses montrent qu’il est egalement necessaire de distinguer entre la 
question de la legitimite de lagence et celle de l’acceptabilite de ses resultats par ses 
partenaires. 

3.2 EVALUATION PAR LES PAIRS 

Quelles que soient les procedures d’assurance qualite utilisees (audit, evaluation 
institutionnelle, accreditation, etc), 1’evaluation par les pairs est le pivot de revaluation. 
Elle cherche a rendre revaluation la plus objective et satisfaisante possible. 

La notion de pairs est comprise differemment par les agences, qui en ont une 
conception plus ou moins large. Dans certains cas, les pairs sont uniquement des 
professeurs d’universite ; dans d autres cas, il peut s’agir de toute personne dont le 
domaine personnel d expertise est pertinent compte tenu de revaluation menee. La 
participation des etudiants aux procedures d evaluation amene d’autres questions sur la 
legitimite de ceux qui participent a lelaboration des appreciations et des avis. 

En definitive, la conduite des procedures d evaluation par les pairs varient selon les 
agences et correspondent a des conceptions differentes sur ce qui est acceptable ou non. 
Dans certains cas, par exemple, les pairs font l’expertise, mais un chef de projet de 
l’agence - qui nest pas un pair - ecrit le rapport final ou en verifie la redaction. Dans 
d’autres cas, seuls les pairs sont charges de la redaction des rapports. 

La question de l’objectivite des jugements est aussi cruciale. Devaluation par les pairs 
presente de nombreux points communs avec le jugement academique. Pour cette raison, 
le jugement ne peut etre tenu pour totalement objectif du seul fait de l’application de 
methodes ou de techniques. Il est ainsi tout a fait possible, et acceptable, qu’un panel de 
pairs aboutisse a des conclusions differentes de celles d’un autre panel qui travaillerait 
sur la meme evaluation. Tout ceci justifie le debat contradictoire sur les resultats ou, le 
cas echeant, sur les procedures d’appel. 

3.3 TRANSPARENCE 

La transparence a ete citee par quelques agences comme etant 1 objectif les incitant a 
reflechir sur leur professionnalisme. En bref, plus une agence est transparente dans son 
fonctionnement et plus elle se juge professionnelle. 

Il est interessant de constater que la notion de transparence n’est generalement pas 
abordee a travers une demonstration de principes, mais le plus souvent au moyen dune 


description de nature technique des methodes et des procedures utilisees. C’est une 
notion qui amene naturellement des discours de nature technique. 

Les agences utilisent aussi cette notion pour demontrer leur legitimite. Cependant, 
la transparence des fonctionnements est-elle suffisante pour garantir la legitimite des 
agences aux yeux des parties prenantes ? Les experts ont estime que la transparence 
nest pas forcement une notion positive, car elle peut en effet etre consideree comme 
abusive par le fait quelle pretendrait rendre limpide des mecanismes qui sont de nature 
subjective. 

Certaines agences ont souligne l’absence de criteres pour juger de la qualite de 
la transparence. Comment des lors, etant depourvu de criteres, peut-on juger la 
transparence dune agence, dune methode d evaluation ou dune decision ? Par ailleurs, 
un defaut de transparence signifie-t-il necessairement qu’une pratique ou un jugement 
sont mal fondes ou peu judicieux ? 

Enfin, doit-on supposer que cette absence de criteres s’explique par le peu 
d’importance que les agences accorderaient a la stabilite et a Tenoned explicite des 
valeurs qui sous-tendent la transparence ? La question parait importante. Cette notion 
risque de perdre son sens et sa place dans Tassurance qualite de Tenseignement 
superieur si Ton ne peut Tapprecier a Taide de criteres. 

3.4 RESULTATS 

Cette valeur n’a pas fait Tobjet de developpements detailles dans les reponses. 
Cependant, quelques-unes d’entre elles ont souligne la dimension politique de la notion 
de resultat. 

Les agences ont acquis une partie de leur legitimite du fait de leur mission qui leur 
impose d’accomplir certaines activites, que ce soit conduire des audits, rendre publics 
les resultats dune evaluation au niveau de la discipline ou de letablissement, proteger 
les standards de qualite, etc. 11 est, par definition, difficile pour elles de repenser 
ces activites ou ces missions. Cela met egalement en evidence le fait que les agences 
travaillent habituellement dans un cadre defini par d’autres acteurs (les gouvernements, 
les conferences de presidents d’universite, etc). 

3.5 CONCLUSION SUR LES RESULTATS DU QUESTIONNAIRE 

En conclusion de ce chapitre, les reponses aux quatre notions choisies pour le 
questionnaire insistent sur Tautorite, la legitimite et les pratiques des agences. Aucune 
des reponses n’a evoque le fait que les notions selectionnees netaient peut-etre pas les 
plus pertinentes pour decrire le travail des agences ; aucune n’a denie leur importance. 
Peut-on deduire de ce constat que la legitimite du travail des agences est veritablement 
guidee par ces quatre notions et les valeurs qui les sous-tendent ? 

4. La legitimite comme question centrale 

Letude QCS I a montre Torganisation systemique des mecanismes d’assurance qualite 
et mis en evidence le fait que la qualite est le resultat des interactions entre partenaires : 
etablissements, ministeres, agences, etc. Elle suggerait que, s’il n’y avait pas de 
definition unique de la qualite ou du concept de qualite, en revanche, Tamelioration de 
la qualite etait le resultat dune « coproduction ». 

Letude QCS II amene a s’interroger sur la recevabilite des resultats des evaluations, 
e’est-a-dire sur les conditions a remplir pour que les resultats des evaluations - que ce 


soit des recommandations, des appreciations ou des decisions d’accreditation - soient 
reconnus comme fondes et legitimes. Cette question est cruciale des lors que les 
resultats des evaluations sont au service de la decision et du pilotage des enseignements 
superieurs. 

Les agences se sont attachees a demontrer le bien-fonde de leur travail en evoquant le 
respect du mandat qui leur est confie et la recherche de la justice et de lequite dans la 
preparation des jugements qu’elles emettent. Elies se sont situees d’emblee sur le terrain 
de la vertu - bien faire ce que Ton doit faire - et ont cherche a demontrer leur fiabilite et 
leur credibilite en mettant en avant une definition du professionnalisme fondee sur les 
notions de transparence, d’independance et d’utilite. 

11 est possible que cette tendance a la justification soit en partie liee a la maniere dont 
le questionnaire etait formule, mais les experts ont estime que tel netait pas le cas. Us y 
ont plutot vu le signe dun manque de confiance des agences en elles-memes. 

C’est aussi probablement la marque d’un manque d’habitude des agences a reflechir 
sur elles-memes. Les praticiens hesitent en effet a remettre en cause les cadres qui 
fondent leur raison d’etre et a s’interroger sur le bien-fonde des decisions politiques 
qui concernent leurs activites. Us se definissent resolument comme operateurs et se 
defendent d’empieter sur des domaines dont la responsabilite releve des decideurs et 
du monde academique. 

Les experts, dune maniere un peu provocatrice sans doute - mais c etait l’esprit des 
relations qui se sont instaurees dans le groupe de travail - , interpellent les agences 
sur l’essentiel : leurs resultats. Us les invitent a s’interroger sur la nature des jugements 
qu’elles formulent, sur les notions d’objectivite et de transparence, ainsi que sur la 
dimension politique de l’assurance de la qualite. 

Letude experimentale QCS II engage les agences a se saisir de questions 
fondamentales. 

5. Le respect des procedures et la recherche de transparence ne suffisent pas 

Au-dela de QCS II stricto sensu, la legitimite est aujourd’hui une question d’actualite 6 . 
Elle est essentielle pour les agences. 

Alors que ces dernieres s evertuent a demontrer qu’elles agissent dans la transparence 
et de maniere independante, il apparait que le respect de ces valeurs ne suffit pas 
a conforter leur legitimite. Les decisions d evaluation, qui jouent un role important 
dans l’enseignement superieur, peuvent-elles pretendre etre le resultat de processus 
techniques tout a fait objectifs ? 

Les references europeennes peuvent nous eclairer : elles parlent des conditions 
de la legitimite des resultats de revaluation, examinent la legitimite des objectifs 
de 1’evaluation (referentiel 1) et la legitimite des demarches et des resultats 
(referentiels 2 et 3). Pour autant, le respect strict des references europeennes 
conduit-il automatiquement a des decisions legitimes et acceptables ? Les praticiens de 
revaluation savent d experience qu’on ne peut evaluer une activite que si l’on en connait 
les intentions. Une evaluation procedurale peut-elle rendre compte, par definition, de la 
qualite des resultats ? Peut-on bien evaluer sans prendre en compte les objectifs propres 
de l’entite evaluee ? Quels sont, des lors, les faits ou les mecanismes qui assurent la 
legitimite et la recevabilite des evaluations ? 


Cf. le seminaire du Conseil de l’Europe a Strasbourg, les 18 et 19 septembre 2006 : L’assurance qualite et sa legitimite dans 
l’enseignement superieur : role des etablissements et des autorites nationales. 


6. La nature politique de revaluation 

Les agences ne peuvent pretendre agir en dehors du champ politique. Leurs resultats 
sont utilises par les decideurs et les etablissements pour le pilotage des systemes 
d’enseignement superieur et pour l’appreciation de leur qualite. A ne pas integrer 
cette dimension, les agences ne risquent-elles pas, sous couvert d’independance, d’etre 
simplement instrumentalisees ? Ce positionnement ne rend-il pas le dialogue avec les 
etablissements d’enseignement superieur plus difficile qu’il ne devrait l’etre ? 

7. Pour aller plus loin 

Cette limite de l’approche procedural et cette reconnaissance de l’implication de fait 
dans le champ politique suscitent une reflexion de la part des agences. Les questions 
suivantes pourraient utilement faire l’objet de reflexions systematiques de leur part afin 
de renforcer la qualite du lien qui les unit a leurs partenaires : 

• les agences sont des acteurs des systemes d’enseignement superieur. Elies sont 
independantes, mais sans etre necessairement autonomes. Des lors, quel est 
le point d’equilibre entre la distance necessaire de 1’evaluation (que traduit la 
notion d’independance de revaluation externe) et la proximite qui permet la 
comprehension des enjeux de l’enseignement superieur ? 

• revaluation par les pairs est censee garantir la legitimite de revaluation, puisque 
le systeme est apprecie par ceux qui le connaissent. Cette valeur est, de 

fait, largement partagee par les decideurs et le monde academique. Elle n’est 
toutefois pas exempte de critiques car elle ne suffit pas a assurer la recevabilite 
des evaluations. Des lors, comment penser le lien entre les beneficiaires des 
evaluations et les agences qui les pratiquent ? Comment parler de la relation entre 
les activites des agences et la finalite des actions qu’elles evaluent ? 

• il parait difficile pour les agences de penser 1’evaluation sans s’interroger sur les 
finalites de l’enseignement superieur. Ne sont-elles pas fondees a s’interroger, avec 
les autres acteurs de l’enseignement superieur, sur la nature des resultats dont le 
systeme a reellement besoin ? 

Il semble bien necessaire de poursuivre la reflexion engagee pour eviter que les notions 
de transparence, d’independance et d’objectivite ne jouent un role trop exclusif dans 
l’elaboration des methodes. Les agences pourront ainsi assumer pleinement, dans un 
esprit d’ouverture, leur role dans les systemes d’enseignement superieur. 


Le groupe de pilotage tient a remercier les experts qui l’ont aide lors de l’etude QCS II : 

Monsieur Colin Brooks, professeur a la Higher Education Academy, Royaume-Uni 
Monsieur Licinio Lima, professeur a I’Universite de Minho, Portugal 
Monsieur Philippe Saltel, maitre de conferences a I’Universite de Grenoble II Pierre- 
Mendes-France, France 

Monsieur Alexander Shurbanov, professeur a VUniversite de Sofia, Bulgarie 


Annex A: List of agencies that 
responded to the questionnaire 

Accreditation Council (Germany) 

Austrian Accreditation Council (Austria) 

Fachochschulrat (Austria) 

Agency for Quality Assurance in the Catalan University System (Catalunya) 
Danish Evaluation Institute (Denmark) 

Comite National d’Evaluation (France) 

Hungarian Accreditation Committee (Hungary) 

Accreditation Commission of the Government of the Czech Republic 

(Czech Republic) 

National Accreditation Agency (Russia) 

Quality Assurance Agency for Higher Education (UK) 

Norwegian Agency for Quality Assurance in Education (Norway) 

Swedish National Agency for Higher Education (Sweden) 

Organisation for the accreditation and assurance of quality in higher education 

(Switzerland) 

European University Association - Institutional Evaluation Programme 


Annex B: accompanying letter and the 
questionnaire used for the QCS II 
project 


Gloucester, 25 July 2006 

QUALITY CONVERGENCE STUDY II 


Dear Colleague, 

ENQA is conducting, with the financial support of the European Commission, a second 
phase of the successful Quality Convergence Study (QCS) and is seeking the help of 
ENQA members to provide information to the project steering group. 

The first QCS concluded that mutual trust, knowledge and understanding were the 
key factors in any move towards the convergence of quality assurance processes in 
Europe and the construction of the European Higher Education Area (EHEA). The aim 
of QCS II is to continue a little further along this path by offering the member agencies 
of ENQA an opportunity to reflect together on the values (common or disparate) that 
underpin their activities. 

Attached to this letter you will find an introduction to the objectives and intended 
outcomes of the project, together with a request for information. The request for 
information (on side two of the attachment) is worded as a short questionnaire. 
However, there is no prescribed pro forma for responding to the questions; agencies are 
invited to provide their answers in free form text. There are four questions and we ask 
that the answers to each question do not take more than one side of A4. Each agency’s 
total response, therefore, should be no longer than four sides of A4. Of course, it is 
acceptable for answers to be shorter. 

Contact details are given in the attachment. The project steering group thanks you 
for taking the time to work with us on this project: your responses are crucial to its 
success. We look forward to hearing from you. 


Yours sincerely, 

Peter Williams 
President, ENQA 


Gloucester, 25 July 2006 

To ENQA members from QCS steering group 


QUALITY CONVERGENCE STUDY II: 

An Introduction to the ENQA Pilot Project and a questionnaire to ENQA members 

1. An introduction to the project 

1.1 The first Quality Convergence Study (QCS) concluded with the notion that the 
focal points for convergence of quality assurance processes in Europe and for the 
agencies that are working in this area towards the construction of the European Higher 
Education Area (EHEA) are those of mutual trust, knowledge and understanding. 

1.2 In the end, this main conclusion served to initiate a closer interest in the quality of 
the international and intercultural dialogue which, step by step, is accompanying the 
construction of the EHEA, and the emergence of common or shared cultural aspects of 
our work. Eventually, each member agency of ENQA will have to find a way of showing 
that it can engender trust in the outcomes of the work it carries out. 

1.3. The aim of QCS II is to continue a little further along this path by offering 

the member agencies of ENQA an opportunity to reflect together on the values that 
underpin their activities. 

1.4. The objective is, therefore, by means of consideration of our concrete activities, 
to reflect on how we would define those values that underpin practice. This approach 
is from an epistemological perspective; we hope that it will add another dimension to 
recent work carried out on practice (e.g. Report on European Standards and Guidelines 
from a Nordic Perspective) and on terminology and vocabulary (ENQA workshop: The 
Language of European Quality Assurance). The project is a pilot and the approach taken 
below is, therefore, experimental. 

2. Project methodology 

2.1 Given the time constraints and the experimental nature of the project, the steering 
group 7 decided to work with a limited number of notions. 

2.2 These notions form part of the normal framework for the work of agencies in 
external quality assurance. They are central to the Standards and Guidelines for Quality 
Assurance in the EEtEA (ESG). In the field of external quality assurance, they are the 
equivalent of expressions of academic values such as freedom of speech, open debate 
and transparency. 


Fiona Crozier (QAA), Bruno Curvale (CNE), Fabrice Henard (CNE), Colin Brooks (Higher Education Academy, UK), Licinio Lima 
(University of Minho, Portugal), Philippe Saltel (University of Grenoble II Pierre Mendes-France) 


2-3 The notions that will be used in the project are: 

• Independence (that gives authority to the evaluation activity); 

• Peer review (that ensures that the results of the evaluation process are credible 
and legitimate); 

• Transparency of the method (the objective being to sustain confidence in the 
process); 

• The results (that justify the utility of the agency’s work). 

3. Questionnaire 

The following questions, to which the project steering group would like the ENQA 
members to provide an answer, relate to each of the notions listed above: 

1. In the agency’s context, what are the main values that underpin the notion? 

2. What is the impact of these values on the methods and procedures employed 
by the agency? 

3. How does the agency ensure that its values are respected? 

4. How does the agency solve any problems it may encounter in relation to the 
values it holds? 

So, for example, for the notion of ‘independence’, the ENQA member agencies are asked 
to consider: 

1) the main values that underpin the notion of ‘independence’ in the context of a 
given agency; 

2) how these values in relation to independence impact on the methods and 
procedures the agency uses; 

3) how is it ensured that the values attributed to the notion of independence are 
respected and 

4) how does the agency resolve any problems it may encounter with the values it 
associates with independence. 

The project is not trying to test hypotheses, to seek out correct answers or to elaborate 
definitions. It is aiming to deepen the capacity for collective thinking about the work 
of the ENQA member agencies. 

It is possible that you may consider that the answers to the questions would vary 
between yourselves and the partners/stakeholders with whom you work closely. If this 
is the case, it would be very interesting for the steering group to know about such 
differences. 

This format should be followed for all four notions. The steering group is looking for 
no more than one page on each notion and we do not expect you to provide us with a 
response that is any longer than 4 sides of A4 maximum. 


When you have completed your response, please e-mail it to Fiona Crozier 
(f.crozier@qaa.ac.uk), Bruno Curvale (bruno.curvale@cne-evaluation.fr) and Fabrice 
Henard (fabrice.henard@cne-evaluation.fr) by 30th September 2006. Your response 
will remain confidential to the steering group and will not be published. 

The steering group is very grateful to you for taking the time to respond. If you 
have any questions about the questionnaire, you can e-mail (as above) or call Fiona 
Crozier (+44 1452 557129), Bruno Curvale (+33 1 55 55 68 40) or Fabrice Henard 
(+33 1 55 55 78 42). A report on the outcomes of the project will be available early 
in 2007. 


Yours sincerely, 

Fiona Crozier 

(On behalf of Bruno Curvale and Fabrice Henard) 


